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11.2.6 Stationary and Limiting Distributions 

Here, we would like to discuss long-term behavior of Markov chains. In particular, we would 
like to know the fraction of times that the Markov chain spends in each state as n becomes large. 
More specifically, we would like to study the distributions 

n in) = [P{X n = 0) P(X n = 1) - ] 

as n —> oo. To better understand the subject, we will first look at an example and then provide a 
general analysis. 


Example 11.12 

Consider a Markov chain with two possible states, S = {0, 1}. In particular, suppose that the 
transition matrix is given by 


P = 


l - a 


a 

1 -b 


where a and b are two real numbers in the interval [0, 1] such that 0 < a + b < 2. Suppose that 
the system is in state 0 at time n = 0 with probability a, i.e., 

jt(°) = [ P ( X o = b ^o = 2 )] = [« 1-a], 


where a E [0, 1], 

a. Using induction (or any other method), show that 


b. Show that 


1 

'b a - 

(1-a - b) n 

| 

a 

-a ' 

a + b 

b a 

a + b 

-b 

b 


lim P n 

n —> go 


a + b 


b a 
b a 


c. Show that 
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lim jr<"> 

n—>co 

• Solution 

o a. For n = 1, we have 


b a 

a+b a+b 


P X = 


' 1 - a 
b 


a 

1 -b 


1 

'b 

a ‘ 

1 - a - b 

| 

a 

-a ‘ 

a + b 

b 

a 

a + b 

-b 

b 


Assuming that the statement of the problem is true for n, we can write P n +1 
as 


pn I _ p n p 


1 / 

‘ b a' 


a 

-a ‘ 

\ 

a + b 1 

b a 

+ (1 - a - b) n 

-b 

b 

H 


1 

b 


a 

1 -b 


i 

'b a' 

(1 -a-b) ,7+l 

1 

a 

-a ‘ 

a + b 

b a 

~r 

a + b 

-b 

b 


which completes the proof. 

b. By assumption 0 < a + b < 2, which implies -1 < 1 - a - b < 1. Thus, 

lim (1 - a - b) n = 0. 

n —> go 

Therefore, 


c. We have 


lim P n = 


a + b 


b a 
b a 
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lim = lim n^^P n 

n —» oo n —» oo 

= n ( 0) lim P" 

/? —> GO 


[a 


1 - a 


a + b 


a 

a 


b a 

a+b a+b 


In the above example, the vector 


lim 

n —> go 


b a 

a+b a+b 


is called the limiting distribution of the Markov chain. Note that the limiting distribution does 
not depend on the initial probabilities a and 1 - a. In other words, the initial state (X (} ) does not 

matter as n becomes large. Thus, for i = 1,2, we can write 


lim P(X n = 01 X () = i) 


b 

a + b' 


lim P(X n = 1 \X Q = i) 

n —> go 


a 

a + b 


(ft') 

Remember that we show P(X n =j\X 0 = i ) by P-,- , which is the entry in the z'th row and /th 
column of P". 

Limitin g Distributions 


The probability distribution n = [n {) , n x , n 2 , •••] is called the limiting distribution of the Markov 
chain X n if 


Kj= lim P(X n =j\X {) = i) 
« —> 00 


for all i,j e .S’, and we have 
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By the above definition, when a limiting distribution exists, it does not depend on the initial state 
(X 0 = i ), so we can write 


iij = lim P(X n = /), for all j E S. 

n —> go 

So far we have shown that the Markov chain in Example 11.12 has the following limiting 
distribution: 


o 

II 

' b 

a 

a + b 

a + b 



. 


Let's now look at mean return times for this Markov chain. 


Example 11.13 

Consider a Markov chain in Example 11.12 : a Markov chain with two possible states, S = {0, 1} 
, and the transition matrix 


P = 


1 - a 


a 

1 -b 


where a and b are two real numbers in the interval [0, 1] such that 0 < a + b < 2. Find the mean 
return times, r {) and r,, for this Markov chain. 


• Solution 


o We can use the method of the law of total probability that we explained before to 
find the mean return times ( Example 11.11 ). We can also find r 0 and r 1 directly as 

follows: 


Let R be the first return time to state 0, i.e., r 0 = E[R |X 0 = 0]. IfX 0 = 0, then 
X l = 0 with probability 1 - a, andXj = 1 with probability a. Thus, using the law of 
total probability, and assuming X 0 = 0, we can write 


r 0 = E[R |Xj = 0]P(X, = 0) + E[R\X x = 1 ]P(X ] = 1) 
= E[R |Xj = 0] • (1 - a) + E[R |Xj = 1] • a. 

If Xj = 0, then R = 1, so 

E[R\X l = 0] = 1. 

If X| = 1, then R ~ 1 + Geometric(b), so 
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E[R |Xj = 1] = 1 + E[Geornetric(b)] 

1 

= 1 + b' 

We conclude 

r 0 = E[R \X x = 0]P(X! = 0) + E[R \X { = 1 }P(X x = 1) 

= 1 • (1 — a) + + 

a + b 
b ' 

Similarly, we can obtain the mean return time to state 1: 

a + b 

r \ = -• 

a 

We notice that for this example, the mean return times are given by the inverse of 
the limiting probabilities. In particular, we have 

1 1 

r 0=~, r \ = — ■ 

n i 

As we will see shortly, this is not a coincidence. In fact, we can explain this 

l 

intuitively. The larger the n i is, the smaller the r- will be. For example, if 7r i = we 
conclude that the chain is in state i one-fourth of the time. In this case, r ■ = 4, which 
means that on average it takes the chain four time units to go back to state i. 


The two-state Markov chain discussed above is a "nice" one in the sense that it has a well- 
defined limiting behavior that does not depend on the initial probability distribution (PMF of A () 
). However, not all Markov chains are like that. For example, consider the same Markov chain; 
however, choose a = b = 1. In this case, the chain has a periodic behavior, i.e., 


In particular, 


X n+2 =X n , for all n. 


X () if n is even 

X 1 if n is odd 
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In this case, the distribution oiX n does not converge to a single PMF. Also, the distribution of 
X n depends on the initial distribution. As another example, if we choose a = b = 0, the chain 
will consist of two disconnected nodes. In this case, 

X n = X Q , for all n. 

Here again, the PMF of X n depends on the initial distribution. Now, the question that arises here 

is: when does a Markov chain have a limiting distribution (that does not depend on the initial 
PMF)? We will next discuss this question. We will first consider finite Markov chains and then 
discuss infinite Markov chains. 

Finite Markov Chains: 

Here, we consider Markov chains with a finite number of states. In general, a finite Markov 
chain can consist of several transient as well as recurrent states. As n becomes large the chain 
will enter a recurrent class and it will stay there forever. Therefore, when studying long-run 
behaviors we focus only on the recurrent classes. 

If a finite Markov chain has more than one recurrent class, then the chain will get absorbed in 
one of the recurrent classes. Thus, the first question is: in which recurrent class does the chain 
get absorbed? We have already seen how to address this when we discussed absorption 
probabilities (see Section 11.2.5 . and Problem 2 of in Section 11.2.7 ). 

Thus, we can limit our attention to the case where our Markov chain consists of one recurrent 
class. In other words, we have an irreducible Markov chain. Note that as we showed in Example 
11.7 . in any finite Markov chain, there is at least one recurrent class. Therefore, in finite 
irreducible chains, all states are recurrent. 

It turns out that in this case the Markov chain has a well-defined limiting behavior if it is 
aperiodic (states have period 1). How do we find the limiting distribution? The trick is to find a 
stationary distribution. Here is the idea: If n = [zr t , n 2 , •■•] is a limiting distribution for a Markov 
chain, then we have 


n = lim 7 r ^ ^ 

n —> go 

= lim |)r (0 V]. 

/2 —> GO 


Similarly, we can write 
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n = lim n n +1 -* 

W —► 00 

= lim [tt (0 V + 1 ] 

« —► 00 

= lim [?r (0} P n P ] 

= [ lim ^ (0) P”]P 

W—► 00 

= ttP. 

We can explain the equation n = nP intuitively: Suppose that X n has distribution n. As we saw 
before, nP gives the probability distribution of X n + l . If we have n = nP, we conclude that X n 
and X n + 1 have the same distribution. In other words, the chain has reached its steady-state 
(limiting) distribution. We can equivalently write n = nP as 

n j = X n k P kp for ally e 5. 
kes 

The righthand side gives the probability of going to state j in the next step. When we equate both 
sides, we are implying that the probability of being in state j in the next step is the same as the 
probability of being in state j now. 

Example 11.14 

Consider a Markov chain in Example 11.12 : a Markov chain with two possible states, S = {0, 1} 

, and the transition matrix 


P = 


1 - a 


a 

1 -b 


where a and b are two real numbers in the interval [0, 1] such that 0 < a + b < 2. Using 

n = nP, 

find the limiting distribution of this Markov chain. 

• Solution 

o Let n = [7r 0 , ttj]. Then, we can write 


[tTq, 7Tj] = nP 


n ~a 


\x () , n j] 


b 


a 

1 -b 


j" 7 t 0 (1 - a) + U\b 7r 0 a + 7r,(1 - b ) J 
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We obtain two equations; however, they both simplify to 

KqU k ^b. 

We remember that n must be a valid probability distribution, i.e., x 0 + n x = 1. Thus, 
we can obtain a unique solution, i.e., 


n = [ 71 \ j 


b a 

a+b a+b 


which is the same answer that we obtained previously. 


We now summarize the above discussion in the following theorem. 

Consider a finite Markov chain {. X n , n = 0, 1, 2,... } where X n E S = {0, 1, 2, •••, r}. Assume 
that the chain is irreducible and a periodic . Then, 


1. The set of equations 


n = nP, 


I 

jes 



has a unique solution. 

2. The unique solution to the above equations is the limiting distribution of the Markov 
chain, i.e., 


uj= lim P(X n = j | X () = i). 


for all i,j E S. 

3. We have 


1 

r- = —, for all j E S, 
J K; 


where r - is the mean return time to state j. 


In practice, if we are given a finite irreducible Markov chain with states 0, 1, 2, •••, r, we first 
find a stationary distribution. That is, we find a probability distribution n that satisfies 


ttj = X *k p kr for all 2 e S, 
keS 



j£S 
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In this case, if the chain is also aperiodic, we conclude that the stationary distribution is a 
limiting distribution. 


Example 11.15 

Consider the Markov chain shown in Figure 11.14. 

1 

2 

1 
4 


1 

2 

Figure 11.14- A state transition diagram. 

a. Is this chain irreducible? 

b. Is this chain aperiodic? 

c. Find the stationary distribution for this chain. 

d. Is the stationary distribution a limiting distribution for the chain? 

• Solution 

o a. The chain is irreducible since we can go from any state to any other states in a 
finite number of steps. 

b. Since there is a self-transition, i .e.,p n > 0, we conclude that the chain is 
aperiodic. 

c. To find the stationary distribution, we need to solve 

1 1 1 

n \ = 4^1 + 3^2 + 2 ^ 3 ’ 

1 

n 2 = 2 ^ 1 ’ 

1 2 1 

n 3 = 4^1 + 3^2 + 2 ^ 3 ’ 

+ TT-r + ^3 = 1. 

We find 

3 3 7 

8 ’ 712 = 16’ = W 
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d. Since the chain is irreducible and aperiodic, we conclude that the above 
stationary distribution is a limiting distribution. 


Countably Infinite Markov Chains: 

When a Markov chain has an infinite (but countable) number of states, we need to distinguish 
between two types of recurrent states: positive recurrent and null recurrent states. 

Remember that if state i is recurrent, then that state will be visited an infinite number of times 
(any time that we visit that state, we will return to it with probability one in the future). We 
previously defined r. as the expected number of transitions between visits to state i. Consider a 

recurrent state i. If r i < oo, then state i is a positive recurrent state. Otherwise, it is called null 
recurrent. 

Let i be a recurrent state. Assuming X Q = i, let R t be the number of transitions needed to return 
to state i, i.e., 


= min {n > l:X n = /}. 

If r- = E[R f \Xq = i] < oo, then i is said to be positive recurrent. If E[R j \X Q = i] = oo, then i is 

said to be null recurrent. 

Theorem 11.2 

Consider an infinite Markov chain {X n , n = 0, 1, 2,... } where X n E S = {0, 1, 2, •••}. Assume 
that the chain is irreducible and a periodic . Then, one of the following cases can occur: 

1. All states are transient , and 

lim P(X n = j | X () = i ) = 0, for all i,j. 

n —> go 

2. All states are null recurrent , and 

lim P(X n = j | X () = i) = 0, for all i,j. 

n —> oo 

3. All states are positive recurrent . In this case, there exists a limiting distribution, 
n = [7r 0 , 7 Tj, •••], where 

tzj = lim P(X n =j\X 0 = i ) > 0, 

n —> go 

for all ij e S. The limiting distribution is the unique solution to the equations 
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n j = Z n k P kp forj = 0, 1, 2, •••, 

k=0 

00 

Z^= !• 

7 = 0 

We also have 

1 

r=—, for ally = 0, 1,2,--, 

J n i 

where r - is the mean return time to state j. 

How do we use the above theorem? Consider an infinite Markov chain {X n , n = 0, 1, 2, 
where X n G S = {0, 1, 2, •••}. Assume that the chain is irreducible and aperiodic. We first try to 
find a stationary distribution n by solving the equations 


OO 

n j = Z n k P kp for/= 0,1,2,-, 

k = 0 


OO 



If the above equations have a unique solution, we conclude that the chain is positive recurrent 
and the stationary distribution is the limiting distribution of this chain. On the other hand, if no 
stationary solution exists, we conclude that the chain is either transient or null recurrent, so 


lim P(X n = j | X () = i ) = 0, for all i,j. 

n —> go 


Example 11.16 

l 

Consider the Markov chain shown in Figure 11.15. Assume that 0 <P<~ 2 . Does this chain have 
a limiting distribution? 


V V V 



1 —p 1 —p 1 —p 


Figure 11.15 - A state transition diagram. 


• Solution 

o This chain is irreducible since all states communicate with each other. It is also 
aperiodic since it includes a self-transition, Pqq > 0. Let's write the equations for a 
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stationary distribution. For state 0, we can write 

= C 1 ~P) n o + d ~P) n l’ 

which results in 


For state 1, we can write 


*1 1 -p K °' 


n \ = P K 0 + d ~p) n 2 
= (1 ~P)n i + (1 ~P)n 2 ’ 


which results in 


n i. 


^2 i /l 1 

z 1 -/? 1 


Similarly, for any j E {1,2, •••}, we obtain 

Kj ~ - 1 5 

P 1 

where a = ~—. Note that since 0 < p < - we conclude that 0 < a < 1. We obtain 

1 -p 1 2 


Kj = a J K () , for / =1,2, 


Finally, we must have 


1 = Z*/ 


7 = 0 


= J^a'jr 0 , 
7 = 0 

1 


i 0 

1 - a u 


(where 0 < a < 1) 


7r 0 (geometric series). 


Thus, k q = l - a. Therefore, the stationary distribution is given by 


Uj = (1 - a)oJ, for j = 0, 1, 2, 


Since this chain is irreducible and aperiodic and we have found a stationary 
distribution, we conclude that all states are positive recurrent and n = [7r 0 , 7 t t , ••■] is 
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the limiting distribution. 


<— previous 
next —* 
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